Display control device, and display control method

ABSTRACT

A yawing information acquisition unit acquires a yaw angle of a vehicle. A deviation possibility prediction unit predicts a possibility of deviation of the vehicle from an expected traveling route, using at least one piece of information among line-of-sight information about an occupant, utterance information, and traffic information. A yawing change prediction unit determines deviation of the vehicle from the expected traveling route, using the yaw angle and the possibility of deviation. In a case where the yawing change prediction unit has determined the deviation, an image generation unit changes a superimposition target and corrects the difference in position between the superimposition target after the change and a display object.

TECHNICAL FIELD

The present invention relates to a display control device and a displaycontrol method for controlling a display device for vehicles.

BACKGROUND ART

A head-up display (hereinafter referred to as “HUD”) for vehicles is adisplay device that enables an occupant to visually recognize imageinformation without significantly moving the line of sight from thefront field of view. In particular, AR-HUD devices that use augmentedreality (AR) can present information to occupants in a more intuitiveand easier-to-understand manner than existing display devices bysuperimposing image information such as a route guide arrow on theforeground such as a road (see Patent Literature 1, for example).

CITATION LIST Patent Literature

Patent Literature 1: JP 2018-140714 A

SUMMARY OF INVENTION Technical Problem

In a case where an AR-HUD device displays a display object (a routeguide arrow, for example) on a superimposition target (an intersection,for example) in the foreground, it is necessary to correct a differencein position between the superimposition target and the display object.The AR-HUD device can provide an occupant with easy-to-understanddisplay by making the occupant visually recognize the superimpositiontarget and the display object as if they were superimposed on eachother. If the display object deviates from the superimposition targetand is superimposed on another superimposition target in the foregroundor is displayed in an empty space, there is a possibility that theoccupant may visually recognize erroneous information. Therefore, it isimportant for an AR-HUD device to correct a difference in displayposition.

Differences in display position related to an AR-HUD device can beclassified into those in the vertical direction and those in thehorizontal direction. The cause of a difference in position between thesuperimposition target and the display object in the vertical directionis primarily a change in the road shape. For example, in a case wherethe position of the superimposition target is higher than the positionof the vehicle due to a slope of a road, the AR-HUD device needs tocorrect the position of the display object upward. Further, in a casewhere the vehicle vibrates in the vertical direction due to unevennessof the road, the AR-HUD device needs to correct the position of thedisplay object in the vertical direction in accordance with thesuperimposition target moving up and down due to the vibration of thevehicle. Patent Literature 1 teaches correction of a difference indisplay position in the vertical direction, but does not teachcorrection of a difference in display position in the horizontaldirection.

The cause of a difference in position between the superimposition targetand the display object in the horizontal direction is primarily thedriver's vehicle operation. The driver drives a vehicle along anexpected traveling route, while moving the steering wheel to right andleft, and adjusting the tilt of the yaw direction of the vehicle.Therefore, even if the vehicle continues to travel in the same lane, theposition of the superimposition target with respect to the vehicle inthe horizontal direction changes. Therefore, the AR-HUD device needs tocorrect the difference in display position in the horizontal directionso that the display object continues to be superimposed on thesuperimposition target.

In a case where a conventional AR-HUD device corrects a difference indisplay position so that the display object continues to be superimposedon the superimposition target, the correction is performed on the basisof the premise that the vehicle travels in an expected traveling route,even if the vehicle sways right and left due to the driver's vehicleoperation. Therefore, in a case where the vehicle moves by a greateramount than the horizontal sway caused by the driver's vehicleoperation, such as at times of a lane change and obstacle avoidance,problems (1), (2), and (3) described below will occur.

(1) The display object moves in the direction opposite from thetraveling direction of the vehicle, which might hinder the driving.

(2) Part or whole of the display object is not displayed in the displayarea of the AR-HUD device, and the information originally indicated bythe display object cannot be conveyed to the occupants.

(3) When the superimposition target is changed as the vehicle keepstraveling, the display object suddenly moves onto a superimpositiontarget after the change, which might hinder the driving.

To prevent problems such as (1), (2), and (3) mentioned above, it isnecessary to detect deviation of the vehicle from the expected travelingroute, and start correcting the difference in display position in thehorizontal direction between the display object and the superimpositiontarget at the start of the deviation. However, in a case where only achange in the yaw angle or the yaw rate of the vehicle is used as in aconventional AR-HUD device, it is difficult to detect the start ofdeviation of the vehicle from the expected traveling route.

The present invention has been made to solve the above problems, andaims to detect deviation of a vehicle from an expected traveling route,and correct a difference in position in the horizontal direction betweena display object and a superimposition target in a case where thevehicle has deviated from the expected traveling route.

Solution to Problem

A display control device according to the present invention is a displaycontrol device that controls a display device that superimposes anddisplays a display object on a superimposition target ahead of avehicle. The display control device includes: a yawing informationacquisition unit that acquires yawing information, the yawinginformation being a yaw angle that is an angle of an actual travelingdirection of the vehicle with respect to an expected traveling route onwhich the vehicle is to travel, or a yaw rate that is an amount ofchange in the yaw angle per unit time; a deviation possibilityprediction unit that predicts a possibility of deviation of the vehiclefrom the expected traveling route, using at least one piece ofinformation among line-of-sight information about an occupant of thevehicle, utterance information about the occupant, and trafficinformation; a yawing change prediction unit that detects deviation ofthe vehicle from the expected traveling route, using the yawinginformation acquired by the yawing information acquisition unit, and thepossibility of deviation predicted by the deviation possibilityprediction unit; and an image generation unit that changes thesuperimposition target and corrects a difference in position between thesuperimposition target after the change and the display object, when theyawing change prediction unit detects deviation of the vehicle from theexpected traveling route.

Advantageous Effects of Invention

According to the present invention, deviation of a vehicle from anexpected traveling route is detected with the use of a yaw angle or ayaw rate, and a deviation possibility predicted from at least one pieceof information among line-of-sight information about an occupant of thevehicle, utterance information, and traffic information. In a case wheredeviation has been detected, the superimposition target is changed, andthe difference in position between the superimposition target after thechange and the display object is corrected. Thus, it is possible tocorrect the difference in position in the horizontal direction betweenthe display object and the superimposition target in a case where thevehicle has deviated from the expected traveling route.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing the relevant parts of a display systemaccording to a first embodiment.

FIG. 2 is a configuration diagram of the display system according to thefirst embodiment when installed in a vehicle.

FIG. 3 is a diagram for explaining yawing information.

FIG. 4 is a flowchart showing an example operation of a display controldevice according to the first embodiment.

FIG. 5A is a diagram showing the foreground from the driver's viewpointbefore a lane change, and illustrates a reference example forfacilitating understanding of the display system according to the firstembodiment.

FIG. 5B is a diagram showing the foreground from the driver's viewpointduring the lane change, and illustrates the reference example forfacilitating understanding of the display system according to the firstembodiment.

FIG. 5C is a diagram showing the foreground from the driver's viewpointafter the lane change, and illustrates the reference example forfacilitating understanding of the display system according to the firstembodiment.

FIG. 6A is a diagram showing the foreground from the driver's viewpointbefore a lane change in the display system according to the firstembodiment.

FIG. 6B is a diagram showing the foreground from the driver's viewpointduring the lane change in the display system according to the firstembodiment.

FIG. 6C is a diagram showing the foreground from the driver's viewpointafter the lane change in the display system according to the firstembodiment.

FIG. 7 is a bird's-eye view showing the situations illustrated in FIGS.5A and 6A.

FIG. 8 is a bird's-eye view showing the situation illustrated in FIG.5B.

FIG. 9 is a bird's-eye view showing the situation illustrated in FIG.6B.

FIG. 10 is a bird's-eye view showing the situations illustrated in FIGS.5C and 6C.

FIG. 11 is a chart showing changes in yawing at a time of deviation froman expected traveling route, and is a reference example for facilitatingunderstanding of the display system according to the first embodiment.

FIG. 12 is a chart showing changes in yawing at a time of deviation froman expected traveling route in the display system according to the firstembodiment.

FIG. 13 is a diagram showing an example hardware configuration of thedisplay system according to the first embodiment.

FIG. 14 is a diagram showing another example hardware configuration ofthe display system according to the first embodiment.

DESCRIPTION OF EMBODIMENTS

To explain the present invention in greater detail, modes for carryingout the invention are described below with reference to the accompanyingdrawings.

First Embodiment

FIG. 1 is a block diagram showing the relevant parts of a display systemaccording to a first embodiment. FIG. 2 is a configuration diagram ofthe display system according to the first embodiment when installed in avehicle. As shown in FIGS. 1 and 2, a vehicle 1 is equipped with adisplay system including a display control device 2 and a display device3, and an information source device 4. The display control device 2generates image information about a display object, and the displaydevice 3 projects display light of the image information onto awindshield 300, so that the driver can visually recognize a displayobject 201 in a virtual image 200 from the position of this driver's eye100 through the windshield 300.

The display control device 2 includes an eye position informationacquisition unit 21, a yawing information acquisition unit 22, adeviation possibility prediction unit 23, a yawing change predictionunit 24, an image generation unit 25, and a virtual image positioninformation acquisition unit 26. The display control device 2 will bedescribed later in detail.

The display device 3 includes an image display unit 31, a reflectivemirror 32, and a reflective mirror adjustment unit 33.

The image display unit 31 outputs display light of image informationgenerated by the image generation unit 25 toward the reflective mirror32. The image display unit 31 is a display such as a liquid crystaldisplay, a projector, or a laser light source. Note that, in a casewhere the image display unit 31 is a liquid crystal display, a backlightis necessary.

The reflective mirror 32 reflects display light output by the imagedisplay unit 31, and projects the display light onto the windshield 300.

The reflective mirror adjustment unit 33 adjusts the tilt angle of thereflective minor 32, to change the reflection angle of the display lightoutput by the image display unit 31 and adjust the position of thevirtual image 200. The reflective mirror adjustment unit 33 outputsreflective minor angle information indicating the tilt angle of thereflective minor 32, to the virtual image position informationacquisition unit 26. In a case where the reflective mirror 32 ismovable, the region in which the driver can visually recognize thevirtual image 200 can be changed depending on the position of thedriver's eye 100, and accordingly, the reflective minor 32 can be madesmaller than a fixed type. Note that the angle adjusting methodimplemented by the reflective minor adjustment unit 33 may be awell-known technique, and therefore, explanation thereof is not madeherein.

The windshield 300 is the surface onto which the virtual image 200 isprojected. The projection target surface is not necessarily thewindshield 300, but may be a semi-reflective mirror called a combiner orthe like. That is, the display device 3 is not necessarily a HUD thatuses the windshield 300, but may be a combiner-type HUD, a head-mounteddisplay (HMD), or the like. As described above, the display device 3 maybe any display device that superimposes and displays the virtual image200 on the foreground of the vehicle 1.

The information source device 4 includes an in-vehicle camera 41, anoutside camera 42, an electronic control unit (ECU) 43, a globalpositioning system (GPS) receiver 44, a navigation device 45, a radarsensor 46, a wireless communication device 47, and an in-vehiclemicrophone 48. This information source device 4 is connected to thedisplay control device 2.

The in-vehicle camera 41 is a camera that captures an image of anoccupant of the vehicle 1 corresponding to the observer of the virtualimage 200. The display system of the first embodiment is provided onassumption that the occupant corresponding to the observer of thevirtual image 200 is the driver. Therefore, the in-vehicle camera 41captures an image of the driver.

The outside camera 42 is a camera that captures an image of thesurroundings of the vehicle 1. For example, the outside camera 42captures an image of a lane in which the vehicle 1 is traveling(hereinafter referred to as the “driving lane”), and an obstacle such asanother vehicle present in the vicinity of the vehicle 1.

The ECU 43 is a control unit that controls various operations of thevehicle 1. The ECU 43 is connected to the display control device 2 witha wire harness (not shown), and can communicate freely with the displaycontrol device 2 by a communication method based on the Controller AreaNetwork (CAN) standard. The ECU 43 is connected to various sensors (notshown), and acquires vehicle information regarding various operations ofthe vehicle 1 from the various sensors. The vehicle information includesinformation about vehicle angle, acceleration, angular velocity, vehiclevelocity, steering angle, the blinkers, and the like. The angularvelocity is formed with angular velocity components generated around thethree axes orthogonal to the vehicle 1, which are yaw rate, pitch rate,and roll rate.

The GPS receiver 44 receives a GPS signal from a GPS satellite (notshown), and calculates position information corresponding to thecoordinates indicated by the GPS signal. The position informationcalculated by the GPS receiver 44 corresponds to current positioninformation indicating the current position of the vehicle 1.

The navigation device 45 corrects the current position informationcalculated by the GPS receiver 44, on the basis of the angular velocityacquired from the ECU 43. The navigation device 45 sets the correctedcurrent position information as the place of departure, and searches forthe traveling route of the vehicle 1 from this place of departure to adestination set by the occupant, using map information stored in astorage device (not shown). Note that, in FIG. 1, the connection linebetween the navigation device 45 and the GPS receiver 44, and theconnection line between the navigation device 45 and the ECU 43 are notshown. The navigation device 45 outputs route guidance information to beused in the traveling route guidance to the display control device 2,and causes the display device 3 to display the route guidanceinformation. The route guidance information includes the travelingdirection of the vehicle 1 at the guidance point (an intersection, forexample) on the traveling route, an estimated time of arrival at awaypoint or the destination, and traffic congestion informationregarding the traveling route and the surrounding roads.

Note that the navigation device 45 may be an information device mountedin the vehicle 1, or may be a mobile communication terminal such as aportable navigation device (PDN) or a smartphone brought into thevehicle 1.

The radar sensor 46 detects the direction and the shape of an obstaclepresent in the vicinity of the vehicle 1, and the distance between thevehicle 1 and the obstacle. The radar sensor 46 includes aradio-frequency sensor in a millimeter waveband, an ultrasonic sensor,or an optical radar sensor, for example.

The wireless communication device 47 acquires various kinds ofinformation by communicating with a network outside the vehicle. Thewireless communication device 47 is formed with a transceiver mounted inthe vehicle 1, or a mobile communication terminal such as a smartphonebrought into the vehicle 1, for example. The network outside the vehicleis the Internet, for example. The various kinds of information to beacquired by the wireless communication device 47 includes weatherinformation about the area around the vehicle 1, or facility informationand the like.

The in-vehicle microphone 48 is a microphone installed in the interiorof the vehicle 1. The in-vehicle microphone 48 collects conversations orutterances of occupants including the driver, and outputs them asutterance information.

Next, each component of the display control device 2 is described.

The eye position information acquisition unit 21 acquires eye positioninformation indicating the position of the driver's eye 100, andline-of-sight information indicating the direction of the line of sight.For example, the eye position information acquisition unit 21 analyzesan image captured by the in-vehicle camera 41, detects the position ofthe driver's eye 100, and sets the detected position of the eye 100 asthe eye position information. The position of the driver's eye 100 maybe the position of each of the driver' left eye and right eye, or may bethe middle position between the left eye and the right eye. Note thatthe eye position information acquisition unit 21 may estimate the middleposition between the right eye and the left eye, from the driver's faceposition in an image captured by the in-vehicle camera 41. The eyeposition information acquisition unit 21 also analyzes the imagecorresponding to the detected position of the driver's eye 100 amongimages captured by the in-vehicle camera 41, and detects the directionof the driver's line of sight.

Note that the eye position information acquisition unit 21 may beincluded in the information source device 4, instead of the displaycontrol device 2. In that case, the eye position information acquisitionunit 21 is formed with a driver monitoring system (DMS) that monitorsthe driver's condition, an occupant monitoring system (OMS) thatmonitors an occupant's condition, or the like.

The yawing information acquisition unit 22 acquires yawing informationindicating the angle of the vehicle 1 in the traveling direction withrespect to an expected traveling route of the vehicle 1. Yawing isrotation of the vehicle 1 about the vertical direction. The yawinginformation is the yaw angle (unit: deg) that is the rotation angle, orthe yaw rate (unit: deg/sec) that is the amount of change in the yawangle per unit time.

The expected traveling route is the traveling route of the vehicle 1,and includes the driving lane and the position of the vehicle 1 in thedriving lane. For example, the yawing information acquisition unit 22acquires route guidance information indicating the traveling directionat the next intersection from the navigation device 45, and calculatesthe traveling route of the vehicle 1 on the basis of the acquired routeguidance information. The yawing information acquisition unit 22 alsoacquires vehicle position information from the GPS receiver 44 and mapinformation including driving lane information from the navigationdevice 45, and, on the basis of these pieces of information, calculatesthe driving lane and the position of the vehicle 1 in the driving lane,for example. Further, the yawing information acquisition unit 22acquires a captured image from the outside camera 42, detects a whiteline or a road shoulder or the like from the acquired image, and, on thebasis of the relationship between the detected white line or the roadshoulder or the like and the vehicle position, calculates the drivinglane and the position of the vehicle 1 in the driving lane, for example.Note that the expected traveling route calculation method implemented bythe yawing information acquisition unit 22 is not limited to the aboveexample. Further, the yawing information acquisition unit 22 may acquireinformation indicating an expected traveling route not calculated by theyawing information acquisition unit 22. In that case, the navigationdevice 45 may independently calculate an expected traveling route, orthe navigation device 45 may calculate an expected traveling route byacquiring information from one of the components of the informationsource device 4, for example.

FIG. 3 is a diagram for explaining yawing information.

In the example shown in FIG. 3, the yawing information acquisition unit22 sets an expected traveling route 401 as the yaw angle reference (0degrees), and acquires a yaw angle that is the angle of the vehicle 1 inthe traveling direction with respect to the expected traveling route401. Alternatively, the yawing information acquisition unit 22calculates a yaw rate, using the acquired yaw angle. The yaw angle andthe yaw rate are positive values in the clockwise direction and arenegative values in the counterclockwise direction with respect to thereference angle (0 degrees). In FIG. 3, a display area 402 correspondsto the area of the windshield 300 onto which the display device 3 canproject the virtual image 200.

For example, the yawing information acquisition unit 22 calculates theyaw angle with respect to the expected traveling route 401, on the basisof the position or tilt of a white line or a road shoulder or the likedetected from an image captured by the outside camera 42. Also, theyawing information acquisition unit 22 calculates the yaw angle bycombining angular velocity detected by a sensor connected to the ECU 43with the vehicle position information or the route guidance informationdescribed above, for example. Alternatively, the yawing informationacquisition unit 22 may calculate a more accurate yaw angle byperforming a statistical processing such as calculation of the meanvalue of yaw angles calculated by a plurality of calculation methods,taking into consideration the imaging cycle of the outside camera 42,the angular velocity detection cycle, the vehicle position acquisitioncycle, and the like, as well as the accuracy of these pieces ofinformation.

The deviation possibility prediction unit 23 predicts the possibility ofdeviation of the vehicle 1 from the expected traveling route 401, on thebasis of the information acquired from the information source device 4.Deviation from the expected traveling route 401 is predicted from atleast one piece of information among route guidance information acquiredfrom the navigation device 45, driving lane information acquired fromthe navigation device 45 or the outside camera 42, obstacle informationacquired from the outside camera 42 or the radar sensor 46, the driver'sline-of-sight information acquired from the eye position informationacquisition unit 21, blinker information acquired from a sensorconnected to the ECU 43, and utterance information regarding an occupantof the vehicle 1 acquired from the in-vehicle microphone 48. Theobstacle information is information indicating the positions ofobstacles that are present around the vehicle 1 and hinder the travelingof the vehicle 1. The driver's line-of-sight information is informationindicating the line-of-sight direction of the driver of the vehicle 1.The blinker information is information indicating whether or not theright blinker and the left blinker of the vehicle 1 are on.

For example, in a case where the route guidance information indicates“turn right at the next intersection”, and the driving lane informationindicates “left lane” or “center lane”, the deviation possibilityprediction unit 23 predicts that there is a high possibility that thevehicle 1 will change lanes to the right lane before entering the nextintersection. For example, in a case where there is an obstacle thathinders the traveling on the expected traveling route 401, such as aparked vehicle ahead of the vehicle 1 in the driving lane, the deviationpossibility prediction unit 23 predicts that there is a high possibilitythat the vehicle 1 will meander to avoid the obstacle. Depending on theposition of the obstacle in the driving lane, the vehicle 1 mightmeander in the driving lane to avoid the obstacle, or the vehicle 1might meander by entering an adjacent lane from the driving lane toavoid the obstacle, and returning to the driving lane after theavoidance. For example, in a case where the driver's line of sight isdirected to a side mirror or backward, the deviation possibilityprediction unit 23 predicts that the vehicle 1 is highly likely tochange lanes. For example, in a case where the blinkers are on, thedeviation possibility prediction unit 23 predicts that the vehicle 1 ishighly likely to change lanes. For example, in a case where an occupantof the vehicle 1 utters “no cars are coming from behind, so change lanesto the right lane”, the deviation possibility prediction unit 23predicts that the vehicle 1 is highly likely to change lanes.

The deviation possibility prediction unit 23 predicts a deviationpossibility by the above prediction method. The deviation possibilityprediction unit 23 may indicate the deviation possibility with adiscrete value at two or more levels, such as “high”, “medium”, and“low”, or with a continuous value from “0%” to “100%”.

For example, in a case where the possibility of a lane change based onthe route guidance information and the driving lane information is low,the deviation possibility prediction unit 23 predicts that the deviationpossibility is “low”. In a case where the possibility of a lane changebased on the route guidance information and the driving lane informationis high, on the other hand, the deviation possibility prediction unit 23predicts that the deviation possibility is “medium”. Further, in a casewhere the possibility of a lane change based on the route guidanceinformation and the driving lane information is high, and thepossibility of a lane change based on the blinker information is high,the deviation possibility prediction unit 23 predicts that the deviationpossibility is “high”. In this manner, the deviation possibilityprediction unit 23 may predict a deviation possibility, depending on thecombination of prediction methods.

For example, in a case where the possibility of a lane change based onthe route guidance information and the driving lane information is high,the deviation possibility prediction unit 23 adds “+30%” to thedeviation possibility. Also, in a case where the possibility of a lanechange based on the blinker information is high, the deviationpossibility prediction unit 23 adds “+30%” to the deviation possibility.In this manner, the deviation possibility prediction unit 23 may addpoints by each predetermined prediction method, to predict a deviationpossibility.

Further, to predict a deviation possibility, the deviation possibilityprediction unit 23 may use past travel history information regarding thevehicle 1. For example, in a case where the route guidance informationindicates “turn right at the next intersection”, and the driving laneinformation indicates “left lane” or “center lane”, the deviationpossibility prediction unit 23 estimates that the frequency of a lanechange to the right lane before the vehicle 1 enters the nextintersection is high, on the basis of the travel history information. Inthis case, the deviation possibility prediction unit 23 adds “+40%”,which is higher than the normal “+30%”, to the deviation possibility.

Alternatively, the deviation possibility prediction unit 23 may predicta deviation possibility, using machine learning or the like.

The yawing change prediction unit 24 predicts a difference in positioncaused between a superimposition display object and the display object201 in the horizontal direction by deviation of the vehicle 1 from theexpected traveling route 401, on the basis of yawing informationacquired by the yawing information acquisition unit 22, and thedeviation possibility predicted by the deviation possibility predictionunit 23. On the basis of the predicted difference in position, theyawing change prediction unit 24 calculates a correction amount forsuperimposed display of the display object 201 on the superimpositiondisplay object. On the basis of the result of the prediction of thedifference in position caused between the superimposition display objectand the display object 201 in the horizontal direction by deviation ofthe vehicle 1 from the expected traveling route 401, the yawing changeprediction unit 24 then instructs the image generation unit 25 to changethe superimposition target on which the display object 201 is to bedisplayed in a superimposed manner, or correct the display mode of thedisplay object 201.

The image generation unit 25 generates image information about thedisplay object 201, and outputs the image information to the imagedisplay unit 31 of the display device 3, to cause the image display unit31 to display this image information. For example, the image generationunit 25 acquires imaging information from the in-vehicle camera 41 andthe outside camera 42, acquires the vehicle position information fromthe GPS receiver 44, acquires vehicle information from various sensorsconnected to the ECU 43, acquires the route guidance information fromthe navigation device 45, acquires the obstacle information from theradar sensor 46, and acquires facility information from the wirelesscommunication device 47. The image generation unit 25 generates imageinformation about the display object 201 indicating the travelingvelocity of the vehicle 1 or a route guide arrow or the like, using atleast one of these acquired pieces of information. Alternatively, theimage generation unit 25 may generate image information about thedisplay object 201 indicating the position of a superimposition targetsuch as the driving lane or an obstacle, or related information aboutthe superimposition target, using at least one of these acquired piecesof information.

At the time of image information generation, the image generation unit25 changes the superimposition target or corrects the display mode ofthe display object 201, on the basis of an instruction from the yawingchange prediction unit 24. In correcting the display mode, the imagegeneration unit 25 corrects the position of the display object 201 onthe basis of the correction amount calculated by the yawing changeprediction unit 24, or switches the display object 201 from a displayedstate to an undisplayed state, for example. The image generation unit 25outputs the generated image information to the image display unit 31.

The display object 201 is an object such as a route guide arrow includedin the image information, and is visually recognized as the virtualimage 200 by the driver. The superimposition target is an object that ispresent in the foreground of the vehicle 1, and the display object 201is to be superimposed on the superimposition target. The superimpositiontarget is the next intersection to which the vehicle 1 is heading,another vehicle or a pedestrian present in the vicinity of the vehicle1, a white line in the driving lane, a facility present in the vicinityof the vehicle 1, or the like.

The image generation unit 25 draws the display object 201 in an image ata position, in size, and with color so that the display object 201 ofthe virtual image 200 appears to be superimposed on the superimpositiontarget, and sets the drawn image as the image information. Note that, ina case where the image display unit 31 can display a binocular parallaximage, the image generation unit 25 may generate a binocular parallaximage as the image information about the display object 201, with thedisplay object being shifted to the right and left in the binocularparallax image.

In a case where the display device 3 has a configuration in which it ispossible to adjust the position of the virtual image 200 by adjustingthe tilt angle of the reflective minor 32 as shown in FIG. 1, the imagegeneration unit 25 acquires virtual image position informationindicating the position of the virtual image 200 depending on the tiltangle of the reflective mirror 32, from the virtual image positioninformation acquisition unit 26. The image generation unit 25 thenchanges the position of the display object 201, on the basis of theacquired virtual image position information.

The virtual image position information acquisition unit 26 acquiresreflective mirror angle information indicating the tilt angle of thereflective mirror 32, from the reflective mirror adjustment unit 33. Thevirtual image position information acquisition unit 26 has a database inwhich the correspondence relationship between the tilt angle of thereflective mirror 32 and the position of the virtual image 200 to bevisually recognized by the driver is defined, for example. By referringto this database, the virtual image position information acquisitionunit 26 identifies the position of the virtual image 200 depending onthe tilt angle of the reflective mirror 32, and outputs the identifiedposition as the virtual image position information to the imagegeneration unit 25.

Note that the virtual image position information acquisition unit 26identifies the position of the virtual image 200 on the basis of thetilt angle of the reflective mirror 32, but the position of the virtualimage 200 may be identified by another method.

Further, in a case where the reflective mirror 32 is fixed, and itsangle is not adjustable, the position of the virtual image 200 may beset beforehand in the virtual image position information acquisitionunit 26 or the image generation unit 25. In a case where the position ofthe virtual image 200 is set beforehand in the image generation unit 25,the virtual image position information acquisition unit 26 is notnecessary.

Next, operations of the display control device 2 are described.

FIG. 4 is a flowchart showing example operations of the display controldevice 2 according to the first embodiment. The display control device 2starts the operation shown in the flowchart in FIG. 4 when the ignitionswitch of the vehicle 1 is turned on, and repeats this operation untilthe ignition switch is turned off, for example.

In step ST1, the display control device 2 acquires various kinds ofinformation from the information source device 4. For example, the eyeposition information acquisition unit 21 acquires a captured image fromthe in-vehicle camera 41, and acquires the driver's eye positioninformation and line-of-sight information, using the acquired capturedimage. Also, the yawing information acquisition unit 22 acquiresinformation from at least one device among the outside camera 42, theECU 43, the GPS receiver 44, and the navigation device 45, andcalculates the expected traveling route 401 and the yawing information,using the acquired information.

In the description below, the yawing information acquisition unit 22calculates a yaw angle as the yawing information.

In step ST2, the deviation possibility prediction unit 23 predicts apossibility of deviation of the vehicle 1 from the expected travelingroute 401, using at least one piece of information among theline-of-sight information, the utterance information, and the trafficinformation acquired from the information source device 4 in step ST1.The traffic information includes at least one piece of information amongthe route guidance information, the driving lane information, and theobstacle information.

If the deviation possibility predicted by the deviation possibilityprediction unit 23 is lower than a predetermined reference (“NO” in stepST2), the yawing change prediction unit 24 performs the operation ofstep ST3. In a case where the deviation possibility is lower than theabove reference, the vehicle 1 is traveling along the expected travelingroute 401. If the deviation possibility predicted by the deviationpossibility prediction unit 23 is equal to or higher than the abovereference (“YES” in step ST2), the yawing change prediction unit 24performs the operation of step ST4. In a case where the deviationpossibility is equal to or higher than the above reference, the vehicle1 is likely to deviate from the expected traveling route 401.

In step ST3, the yawing change prediction unit 24 sets a first thresholdas the yawing information threshold for determining to change thesuperimposition target. Note that the value of the first threshold maybe a fixed value, or may be a variable value that changes with thedeviation possibility or the like. This first threshold is the thresholdfor determining that the vehicle 1 has deviated from the expectedtraveling route 401 and for determining to change the superimpositiontarget, in a situation where the deviation possibility is lower than thereference and the vehicle 1 is traveling along the road shape. In a casewhere the possibility of deviation of the vehicle 1 from the expectedtraveling route 401 is low, the change in the yaw angle of the vehicle 1is small, because the driver drives along the road shape while finelyadjusting the yaw angle of the vehicle 1. In the first embodiment, whenthe yaw angle of the vehicle 1 changes with the driver's operation andthe road shape, the superimposition target is not changed. When the yawangle of the vehicle 1 changes with a lane change or obstacle avoidance,the superimposition target is changed. Therefore, the first threshold isset at a value that is greater than the amount of change caused in theyaw angle of the vehicle 1 by the driver's operation and the road shape,but is smaller than the amount of change caused in the yaw angle of thevehicle 1 by a lane change or obstacle avoidance.

In step ST4, the yawing change prediction unit 24 sets a secondthreshold as the yawing information threshold for determining to changethe superimposition target. Note that the value of the second thresholdmay be a fixed value, or may be a variable value that changes with thedeviation possibility or the like. This second threshold is thethreshold for determining that the vehicle 1 has deviated from theexpected traveling route 401 and for determining to change thesuperimposition target, in a situation where the deviation possibilityis equal to or higher than the reference and the vehicle 1 is likely todeviate from the expected traveling route 401. In a case where thepossibility of deviation of the vehicle 1 from the expected travelingroute 401 is high, the change in the yaw angle of the vehicle 1 becomeslarger at a time when the vehicle 1 changes lanes or avoids an obstacle.In the first embodiment, the absolute value of the second threshold isset at a smaller value than the absolute value of the first threshold,so that a start of a lane change or obstacle avoidance by the vehicle 1can be detected.

Note that, in a case where the first threshold and the second thresholdare variable values, the yawing change prediction unit 24 sets a valuedepending on the driving characteristics of the driver, using the pasttravel history information regarding each driver, for example. Further,the yawing change prediction unit 24 may cause the value to differbetween a lane change and obstacle avoidance.

In step ST5, if the yaw angle acquired from the yawing informationacquisition unit 22 is equal to or greater than the first threshold setin step ST3, or is equal to or greater than the second threshold set instep ST4 (“YES” in step ST5), the yawing change prediction unit 24performs the operation of step ST6. If the yaw angle acquired from theyawing information acquisition unit 22 is smaller than the firstthreshold set in step ST3, or is smaller than the second threshold setin step ST4 (“NO” in step ST5), on the other hand, the yawing changeprediction unit 24 performs the operation of step ST7.

In step ST6, the yawing change prediction unit 24 determines that thevehicle 1 has deviated from the expected traveling route 401, andacquires an expected traveling route 401 on which the vehicle 1 isexpected to travel after the deviation (this route will be hereinafterreferred to as the “expected traveling route 401 after the change”),from the yawing information acquisition unit 22. Further, to correct thedifference in position caused between the display object 201 and thesuperimposition target in the horizontal direction by deviation of thevehicle 1 from the expected traveling route 401, the yawing changeprediction unit 24 instructs the image generation unit 25 to change thesuperimposition target on the basis of the expected traveling route 401after the change, and to correct the display mode of the display object201 to match the changed superimposition target. Note that, in a casewhere the difference between the yaw angle and the first threshold, orthe difference between the yaw angle and the second threshold is equalto or larger than a predetermined value, or where the superimpositiontarget is present outside the display area 402 of the image display unit31 due to deviation of the vehicle 1 from the expected traveling route401, the yawing change prediction unit 24 may instruct the imagegeneration unit 25 to put the display object 201 into an undisplayedstate, instead of to change the superimposition target.

In step ST7, the yawing change prediction unit 24 instructs the imagegeneration unit 25 to correct the display mode of the display object 201so as to eliminate the difference in position caused between the displayobject 201 and the superimposition target in the horizontal direction bythe driver's driving operation or the road shape. Note that, in stepST7, the yawing change prediction unit 24 does not change thesuperimposition target, because the vehicle 1 has not deviated from theexpected traveling route 401.

In step ST8, the image generation unit 25 generates image informationabout the display object 201, using the various kinds of informationacquired from the information source device 4. The image generation unit25 also corrects the display mode such as the position and the size ofthe display object 201 so that the display object 201 is superimposed onthe superimposition target designated by the yawing change predictionunit 24. For example, in a case where the correction amount forcorrecting a difference in position between the superimposition targetand the display object 201 in the horizontal direction is designated,the image generation unit 25 acquires the yaw angle from the yawinginformation acquisition unit 22, and corrects the position of thedisplay object 201, using the acquired yaw angle and the abovecorrection amount. The image generation unit 25 then outputs the imageinformation about the display object 201 to the image display unit 31,to cause the image display unit 31 to project the image information ontothe windshield 300. Note that, in a case where the display device 3 isalready displaying the image information about the display object 201,the image generation unit 25 corrects the display object 201 in theimage information in accordance with an instruction from the yawingchange prediction unit 24.

Next, the difference between a case where there is only one thresholdfor changing the superimposition target and a case where there are twothresholds that are the first threshold and the second threshold isdescribed. In the description below, the case where there is only onethreshold for changing the superimposition target will be referred to asthe “reference example”, and the first threshold will be mentioned asthis threshold.

FIGS. 5A, 5B, and 5C are diagrams showing the foreground from thedriver's viewpoint, and illustrate the reference example forfacilitating understanding of the display system according to the firstembodiment. FIGS. 6A, 6B, and 6C are diagrams showing the foregroundfrom the driver's viewpoint in the display system according to the firstembodiment. FIGS. 5A and 6A each show the foreground from the driver'sviewpoint before a lane change. FIGS. 5B and 6B each show the foregroundfrom the driver's viewpoint during the lane change. FIGS. 5C and 6C eachshow the foreground from the driver's viewpoint after the lane change.

FIG. 7 is a bird's-eye view showing the situations illustrated in FIGS.5A and 6A. FIG. 8 is a bird's-eye view showing the situation illustratedin FIG. 5B. FIG. 9 is a bird's-eye view showing the situationillustrated in FIG. 6B. FIG. 10 is a bird's-eye view showing thesituations illustrated in FIGS. 5C and 6C.

FIG. 11 is a chart showing changes in yawing at a time of deviation fromthe expected traveling route, and is a reference example forfacilitating understanding of the display system according to the firstembodiment. FIG. 12 is a chart showing changes in yawing at a time ofdeviation from the expected traveling route in the display systemaccording to the first embodiment.

Note that, in FIGS. 11 and 12, only the first threshold and the secondthreshold that are positive values are shown, and the first thresholdand the second threshold that are negative values are not shown. Theabsolute value of the positive first threshold and the absolute value ofthe negative first threshold may be the same or may be different.Likewise, the absolute value of the positive second threshold and theabsolute value of the negative second threshold may be the same or maybe different.

First, the reference example is described.

In the reference example, a superimposition target 403 (an intersectionin the center lane, for example) is seen through the windshield 300 inthe display area 402 of the windshield 300, as shown in FIG. 5A. In thedisplay area 402, the display object 201 (a route guide arrow, forexample) that is a virtual image is superimposed and displayed on thesuperimposition target 403. Meanwhile, as shown in FIG. 7, the vehicle 1is traveling in the center lane along the expected traveling route 401(time T0 to time T2 in FIG. 11).

As shown in FIG. 8, the vehicle 1 starts changing lanes from the centerlane to the right lane, to turn right at the intersection in accordancewith the expected traveling route 401 (time T2 in FIG. 11). Because theyaw angle that has changed with the lane change is smaller than thefirst threshold, the display object 201 remains superimposed anddisplayed on the superimposition target 403 as shown in FIG. 5B. On theother hand, the display area 402 moves to the right as the vehicle 1changes lanes. Therefore, in FIG. 5B, the display object 201 moves inthe direction opposite from the traveling direction of the vehicle 1,and might hinder the driving. Also, part of the display object 201 isnot displayed because it is now outside the display area 402. Therefore,there is a possibility that the information originally indicated by thedisplay object 201 cannot be correctly conveyed to the driver.

In a case where the yaw angle becomes equal to or greater than the firstthreshold (time T3 in FIG. 11) after the start of the lane change (timeT2 in FIG. 11), the yawing change prediction unit 24 determines thatdeviation from the expected traveling route 401 has occurred. At thistime T3, the yawing change prediction unit 24 determines that it isnecessary to change the expected traveling route 401 and thesuperimposition target 403. The yawing change prediction unit 24 theninstructs the image generation unit 25 to change the superimpositiontarget 403, on the basis of the expected traveling route 401 after thechange, the amount of change in the yaw angle, and the like. Uponreceipt of the instruction from the yawing change prediction unit 24,the image generation unit 25 changes the superimposition target 403 froman intersection in the center lane, which is the expected travelingroute 401 before the change, to an intersection in the right lane, whichis the expected traveling route 401 after the change, on the basis ofthe amount of change in the yaw angle and the like. Accordingly, thedisplay object 201 is superimposed and displayed on the superimpositiontarget 403, which is an intersection in the right lane, as shown inFIGS. 5C and 10. The vehicle 1 travels in the right lane along theexpected traveling route 401. At time T3 in FIG. 11, the foreground fromthe driver's viewpoint changes from the one shown in FIG. 5B to the oneshown FIG. 5C, and the display object 201 suddenly moves from the centerlane to the right lane, which might hinder the driving.

As described above, in a case where there is only one threshold forchanging the superimposition target, a great value needs to be set asthe threshold, to distinguish a yaw angle change for traveling along theroad shape from a yaw angle change caused by deviation of the vehicle 1from the expected traveling route 401. Therefore, a delay is caused indetermining to change lanes or avoid an obstacle. Further, even if theorientations of the vehicle 1 are the same as shown in FIGS. 8 and 9,the timing to change the expected traveling route 401 is delayed in thecase where there is only one threshold. Therefore, the yaw angles aftertime T2 and T12 are different. As a result, in the case where there isonly one threshold, the difference in display position between thedisplay object 201 and the superimposition target 403 cannot beappropriately corrected, and the visibility of the foreground includingthe display object 201 is degraded.

Next, an example of the first embodiment is described.

In the first embodiment, the superimposition target 403 (an intersectionin the center lane, for example) is seen through the windshield 300 inthe display area 402 of the windshield 300, as shown in FIG. 6A. In thedisplay area 402, the display object 201 (a route guide arrow, forexample) that is a virtual image is superimposed and displayed on thesuperimposition target 403. Meanwhile, as shown in FIG. 7, the vehicle 1is traveling in the center lane along the expected traveling route 401(time T10 to time T12 in FIG. 12).

The deviation possibility prediction unit 23 predicts that the deviationpossibility is high before the intersection, because the vehicle 1 is tochange lanes from the center lane to the right lane to turn right at theintersection in accordance with the expected traveling route 401. As thedeviation possibility is equal to or higher than the reference, theyawing change prediction unit 24 sets the second threshold as thethreshold for determining to change the superimposition target (time T11in FIG. 12).

As shown in FIG. 9, the vehicle 1 starts changing lanes from the centerlane to the right lane, to turn right at the intersection in accordancewith the expected traveling route 401 (time T12 in FIG. 12). As the yawangle that has changed with the lane change quickly becomes equal to orhigher than the second threshold (time T13 in FIG. 12), the yawingchange prediction unit 24 determines that deviation from the expectedtraveling route 401 has occurred. At this time T13, the yawing changeprediction unit 24 determines that it is necessary to change theexpected traveling route 401 and the superimposition target 403. Theyawing change prediction unit 24 then instructs the image generationunit 25 to change the superimposition target 403, on the basis of theexpected traveling route 401 after the change, the amount of change inthe yaw angle, and the like. Further, on the basis of the amount ofchange in the yaw angle and the like, the yawing change prediction unit24 calculates the correction amount for the difference in position inthe horizontal direction between the superimposition target 403 afterthe change and the display object 201, and informs the image generationunit 25 of the correction amount. Upon receipt of an instruction fromthe yawing change prediction unit 24, the image generation unit 25changes the superimposition target 403 from the intersection in thecenter lane to an intersection in the right lane, on the basis of theamount of change in the yaw angle and the like (time T13 in FIG. 12).The image generation unit 25 also corrects the display mode of thedisplay object 201 as shown in FIGS. 6B and 9, on the basis of thecorrection amount designated by the yawing change prediction unit 24.Accordingly, the display object 201 moves in the same direction as thetraveling direction of the vehicle 1, and the display object 201 doesnot move out of the display area 402. Also, the foreground from thedriver's viewpoint changes from the one shown in FIG. 6B to the oneshown in FIG. 6C, and sudden movement of the display object 201 isprevented.

As described above, the display control device 2 according to the firstembodiment includes the yawing information acquisition unit 22, thedeviation possibility prediction unit 23, the yawing change predictionunit 24, and the image generation unit 25. The yawing informationacquisition unit 22 acquires the yaw angle or the yaw rate of thevehicle 1 as yawing information. The deviation possibility predictionunit 23 predicts a possibility of deviation of the vehicle 1 from theexpected traveling route 401, using at least one piece of informationamong line-of-sight information about an occupant of the vehicle 1,utterance information about the occupant, and traffic information. Theyawing change prediction unit 24 detects deviation of the vehicle 1 fromthe expected traveling route 401, using the yawing information and thedeviation possibility. In a case where the yawing change prediction unit24 has detected deviation of the vehicle 1 from the expected travelingroute 401, the image generation unit 25 changes the superimpositiontarget 403, and corrects the difference in position between thesuperimposition target 403 after the change and the display object 201.In this manner, the display control device 2 can detect deviation of thevehicle 1 from the expected traveling route 401, using the yaw angle orthe yaw rate, and the deviation possibility predicted with the use of atleast one piece of information among the line-of-sight information, theutterance information, and the traffic information. The display controldevice 2 can also prevent problems such as the problems (1), (2), and(3) described above, by changing the superimposition target 403 whendeviation is detected, and correcting the difference in position in thehorizontal direction between the superimposition target 403 after thechange and the display object 201. Thus, the display control device 2can correct the difference in position in the horizontal directionbetween the display object 201 and the superimposition target 403 in acase where the vehicle 1 has deviated from the expected traveling route401.

Also, according to the first embodiment, in a case where the deviationpossibility predicted by the deviation possibility prediction unit 23 islower than the predetermined reference (“NO” in step ST2 in FIG. 4), theyawing change prediction unit 24 sets the first threshold as thethreshold for determining to change the superimposition target (step ST3in FIG. 4). In a case where the yawing information acquired by theyawing information acquisition unit 22 is equal to or greater than thefirst threshold (“YES” in step ST5 in FIG. 4), the yawing changeprediction unit 24 then instructs the image generation unit 25 to changethe superimposition target 403 or make the display object 201undisplayed (step ST6 in FIG. 4). Thus, the display control device 2 candetect deviation of the vehicle 1 from the expected traveling route 401even in a case where the deviation possibility is low. Further, thedisplay control device 2 does not make any correction for maintainingthe superimposed display of the superimposition target 403 and thedisplay object 201 before the change at the time of the deviationdetection, but changes the superimposition target 403 or makes thedisplay object 201 undisplayed. Thus, display without any unnaturalnesscan be performed.

Further, according to the first embodiment, in a case where thedeviation possibility predicted by the deviation possibility predictionunit 23 is equal to or higher than the predetermined reference (“YES” instep ST2 in FIG. 4), the yawing change prediction unit 24 sets thesecond threshold as the threshold for determining to change thesuperimposition target (step ST4 in FIG. 4). In a case where the yawinginformation acquired by the yawing information acquisition unit 22 isequal to or greater than the second threshold (“YES” in step ST5 in FIG.4), the yawing change prediction unit 24 then instructs the imagegeneration unit 25 to change the superimposition target 403 or make thedisplay object 201 undisplayed (step ST6 in FIG. 4). As a result, in acase where the deviation possibility is high, the display control device2 can detect a start of deviation of the vehicle 1 from the expectedtraveling route 401, using the second threshold, which is smaller thanthe first threshold. Further, the display control device 2 does not makeany correction for maintaining the superimposed display of thesuperimposition target 403 and the display object 201 before the changeat the time of the deviation start detection, but changes thesuperimposition target 403 or makes the display object 201 undisplayed.Thus, display without any unnaturalness can be performed.

Also, according to the first embodiment, in a case where the deviationpossibility predicted by the deviation possibility prediction unit 23 islower than the predetermined reference (“NO” in step ST2 in FIG. 4), andthe yawing information acquired by the yawing information acquisitionunit 22 is smaller than the first threshold (“NO” in step ST5 in FIG.4), the yawing change prediction unit 24 instructs the image generationunit 25 to correct the difference in position between thesuperimposition target 403 and the display object 201 (step ST7 in FIG.4).

Further, in a case where the deviation possibility predicted by thedeviation possibility prediction unit 23 is equal to or higher than thepredetermined reference (“YES” in step ST2 in FIG. 4), and the yawinginformation acquired by the yawing information acquisition unit 22 issmaller than the second threshold (“NO” in step ST5 in FIG. 4), theyawing change prediction unit 24 instructs the image generation unit 25to correct the difference in position between the superimposition target403 and the display object 201 (step ST7 in FIG. 4).

In either case, the display control device 2 can make correction formaintaining the superimposed display of the display object 201 on thesuperimposition target 403 while the vehicle 1 is traveling along theexpected traveling route 401. Thus, display without any unnaturalnesscan be performed.

Next, a modification of the display control device 2 according to thefirst embodiment is described.

The yawing change prediction unit 24 of the first embodiment uses a yawangle as yawing information, and sets a first threshold and a secondthreshold that match the value of the yaw angle. However, a yaw rate maybe used as yawing information, and the first threshold and the secondthreshold that match the value of the yaw rate may be set. In the caseof this modification, if the yaw rate is equal to or higher than thefirst threshold or the second threshold in step ST5 in FIG. 4, theyawing change prediction unit 24 determines that the vehicle 1 hasdeviated from the expected traveling route 401. In the case where theyaw rate is used, it is possible to detect a start of deviation of thevehicle 1 from the expected traveling route 401 more quickly than in acase where the yaw angle is used, and it might also be possible tochange the superimposition target 403 more quickly.

In a modification of the first embodiment, if the deviation possibilitypredicted by the deviation possibility prediction unit 23 is equal to orhigher than the predetermined reference (“YES” in step ST2 in FIG. 4),that is, if the possibility of deviation of the vehicle 1 from theexpected traveling route 401 is high, the yawing change prediction unit24 instructs the image generation unit 25 to temporarily stop thecorrection of the difference in position between the superimpositiontarget 403 and the display object 201. After that, when thesuperimposition target 403 after a change enters the display area 402,or when the superimposition target 403 after the change and the displayobject 201 come close to each other within a predetermined distance, theyawing change prediction unit 24 instructs the image generation unit 25to resume the correction of the difference in position between thesuperimposition target 403 after the change and the display object 201.In the case of this modification, movement of the display object causedby a change of the superimposition target occurs in accordance with achange in the yaw angle, and thus, display without any unnaturalness canbe performed.

In a modification of the first embodiment, in a case where the deviationpossibility is equal to or higher than the reference, and thepossibility of deviation of the vehicle 1 from the expected travelingroute 401 is high, the yawing change prediction unit 24 predicts thesuperimposition target 403 after a change at that point of time (timeT11 in FIG. 12). Also, in a case where the yawing change prediction unit24 determines that the deviation possibility is equal to or higher thanthe reference, and predicts that the possibility of deviation of thevehicle 1 from the expected traveling route 401 is high, the yawinginformation acquisition unit 22 also predicts the expected travelingroute 401 after a change at that point of time (time T11 in FIG. 12). Inthe case of this modification, the time until the image generation unit25 corrects the display object 201 can be made shorter than in a casewhere the superimposition target 403 and the expected traveling route401 after a change are calculated when the yawing information isdetermined to be equal to or greater than the first threshold or thesecond threshold after the vehicle 1 starts deviating from the expectedtraveling route 401. Note that the superimposition target 403 and theexpected traveling route 401 after the change are predicted with the useof at least one piece of information among yawing information,line-of-sight information, utterance information, traffic information,and other various kinds of information.

In a modification of the first embodiment, the image generation unit 25predicts the position of the vehicle 1 at the time of display of thedisplay object 201, on the basis of vehicle velocity and yawinginformation. The image generation unit 25 corrects the position of thedisplay object 201, taking into consideration the difference between theposition of the vehicle 1 at the time when each component of the displaycontrol device 2 acquires information in step ST1 in FIG. 4, and thepredicted position of the vehicle 1 at the time when image informationis generated and is displayed on the image display unit 31 in step ST8.In the case of this modification, it is possible to correct thedifference in position caused between the display object 201 and thesuperimposition target 403 when the vehicle 1 is traveling.

Note that, in the first embodiment, the display system provided for thedriver has been described as an example. However, the display system maybe provided for an occupant other than the driver.

Also, in the first embodiment, the display device 3 is a HUD, a HMD, orthe like, but may be a center display or the like installed on thedashboard of the vehicle 1. The center display superimposes imageinformation about the display object 201 generated by the imagegeneration unit 25 of the display control device 2, on an image of theforeground of the vehicle 1 captured by the outside camera 42. Asdescribed above, the display device 3 is only required to be capable ofsuperimposing the display object 201 on the foreground of the vehicle 1through the windshield 300 or the foreground captured by the outsidecamera 42.

Lastly, the hardware configuration of the display system according tothe first embodiment is described.

FIG. 13 is a diagram showing an example hardware configuration of thedisplay system according to the first embodiment. In FIG. 13, aprocessing circuit 500 is connected to the display device 3 and theinformation source device 4, and can exchange information. FIG. 14 is adiagram showing another example hardware configuration of the displaysystem according to the first embodiment. In FIG. 14, a processor 501and a memory 502 are both connected to the display device 3 and theinformation source device 4. The processor 501 is capable of exchanginginformation with the display device 3 and the information source device4.

The functions of the eye position information acquisition unit 21, theyawing information acquisition unit 22, the deviation possibilityprediction unit 23, the yawing change prediction unit 24, the imagegeneration unit 25, and the virtual image position informationacquisition unit 26 in the display control device 2 are achieved with aprocessing circuit. That is, the display control device 2 includes aprocessing circuit for achieving the above functions. The processingcircuit may be the processing circuit 500 as dedicated hardware, or maybe the processor 501 that executes a program stored in the memory 502.

In a case where the processing circuit is dedicated hardware as shown inFIG. 13, the processing circuit 500 may be a single circuit, a compositecircuit, a programmed processor, a parallel-programmed processor, anapplication specific integrated circuit (ASICs), a field-programmablegate array (FPGAs), or a combination thereof, for example. The functionsof the eye position information acquisition unit 21, the yawinginformation acquisition unit 22, the deviation possibility predictionunit 23, the yawing change prediction unit 24, the image generation unit25, and the virtual image position information acquisition unit 26 maybe achieved with a plurality of processing circuits 500, or thefunctions of the respective components may be achieved with oneprocessing circuit 500.

In a case where the processing circuit is the processor 501 as shown inFIG. 14, the functions of the eye position information acquisition unit21, the yawing information acquisition unit 22, the deviationpossibility prediction unit 23, the yawing change prediction unit 24,the image generation unit 25, and the virtual image position informationacquisition unit 26 are achieved with software, firmware or acombination of software and firmware. Software or firmware is written asa program, and is stored in the memory 502. The processor 501 achievesthe functions of the respective components by reading and executing theprogram stored in the memory 502. That is, the display control device 2includes the memory 502 for storing a program for eventually carryingout the steps shown in the flowchart in FIG. 4 when executed by theprocessor 501. This program can also be regarded as a program forcausing a computer to carry out or implement the procedures or themethods adopted by the eye position information acquisition unit 21, theyawing information acquisition unit 22, the deviation possibilityprediction unit 23, the yawing change prediction unit 24, the imagegeneration unit 25, and the virtual image position informationacquisition unit 26.

Here, the processor 501 is a central processing unit (CPU), a processingunit, an arithmetic unit, a microprocessor, or the like.

The memory 502 may be a nonvolatile or volatile semiconductor memorysuch as a random access memory (RAM), a read only memory (ROM), anerasable programmable ROM (EPROM), or a flash memory, may be a magneticdisk such as a hard disk or a flexible disk, or may be an optical disksuch as a compact disc (CD) or a digital versatile disc (DVD).

Note that some of the functions of the eye position informationacquisition unit 21, the yawing information acquisition unit 22, thedeviation possibility prediction unit 23, the yawing change predictionunit 24, the image generation unit 25, and the virtual image positioninformation acquisition unit 26 may be achieved with dedicated hardware,and some of these functions may be achieved with software or firmware.For example, the functions of the eye position information acquisitionunit 21 are achieved with dedicated hardware, and the functions of theyawing information acquisition unit 22, the deviation possibilityprediction unit 23, the yawing change prediction unit 24, the imagegeneration unit 25, and the virtual image position informationacquisition unit 26 are achieved with software or firmware. In thismanner, the processing circuit in the display control device 2 canachieve the above functions with hardware, software, firmware, or acombination thereof.

Within the scope of the present invention, modifications may be made toany component of the embodiment, or any component may be omitted fromthe embodiment.

INDUSTRIAL APPLICABILITY

A display control device according to the present invention is designedto correct a difference in position in the horizontal direction betweena display object and a superimposition target, and accordingly, issuitable as a display control device that controls a HUD and the likeinstalled in a vehicle.

REFERENCE SIGNS LIST

1: vehicle, 2: display control device, 3: display device, 4: informationsource device, 21: eye position information acquisition unit, 22: yawinginformation acquisition unit, 23: deviation possibility prediction unit,24: yawing change prediction unit, 25: image generation unit, 26:virtual image position information acquisition unit, 31: image displayunit, 32: reflective mirror, 33: reflective mirror adjustment unit, 41:in-vehicle camera, 42: outside camera, 43: ECU, 44: GPS receiver, 45:navigation device, 46: radar sensor, 47: wireless communication device,48: in-vehicle microphone, 100: driver's eye, 200: virtual image, 201:display object, 300: windshield, 401: expected traveling route, 402:display area, 403: superimposition target, 500: processing circuit, 501:processor, 502: memory

1. A display control device that controls a display device thatsuperimposes and displays a display object on a superimposition targetahead of a vehicle, the display control device comprising: processingcircuitry configured to acquire yawing information, the yawinginformation being a yaw angle that is an angle of an actual travelingdirection of the vehicle with respect to an expected traveling route onwhich the vehicle is to travel, or a yaw rate that is an amount ofchange in the yaw angle per unit time; predict possibility of deviationof the vehicle from the expected traveling route, using at least onepiece of information among line-of-sight information about an occupantof the vehicle, utterance information about the occupant, and trafficinformation; determine deviation of the vehicle from the expectedtraveling route, using the acquired yawing information, and thepredicted possibility of deviation; and change the superimpositiontarget and correct a difference in position between the superimpositiontarget after the change and the display object, when the processingcircuitry determines the deviation of the vehicle from the expectedtraveling route.
 2. The display control device according to claim 1,wherein the traffic information is at least one piece of informationamong route guidance information output by a navigation device, laneinformation regarding a lane in which the vehicle is traveling, andobstacle information regarding an obstacle present around the vehicle.3. The display control device according to claim 1, wherein, when thepredicted possibility of deviation is lower than a predeterminedreference, the yawing change prediction unit processing circuitry sets afirst threshold as a threshold for determining whether to change thesuperimposition target, and, when the acquired yawing information isequal to or greater than the first threshold, the processing circuitrychanges the superimposition target or make the display objectundisplayed.
 4. The display control device according to claim 3,wherein, when the predicted possibility of deviation is equal to orhigher than the predetermined reference, the processing circuitry sets asecond threshold as the threshold for determining whether to change thesuperimposition target, the second threshold being smaller than thefirst threshold, and, when the acquired yawing information is equal toor greater than the second threshold, the processing circuitry changesthe superimposition target or make the display object undisplayed. 5.The display control device according to claim 3, wherein, when thepredicted possibility of deviation is lower than the predeterminedreference, and the acquired yawing information is smaller than the firstthreshold, the processing circuitry corrects a difference in positionbetween the superimposition target and the display object.
 6. Thedisplay control device according to claim 4, wherein, when the predictedpossibility of deviation is equal to or higher than the predeterminedreference, and the acquired yawing information is smaller than thesecond threshold, the processing circuitry corrects a difference inposition between the superimposition target and the display object. 7.The display control device according to claim 4, wherein, when thepredicted possibility of deviation is equal to or higher than thepredetermined reference, the processing circuitry does not correct adifference in position between the superimposition target and thedisplay object, and, when the superimposition target after the changeand the display object come within a predetermined distance from eachother, the processing circuitry corrects a difference in positionbetween the superimposition target after the change and the displayobject.
 8. A display control method for controlling a display devicethat superimposes and displays a display object on a superimpositiontarget ahead of a vehicle, the display control method comprising:acquiring yawing information, the yawing information being a yaw anglethat is an angle of an actual traveling direction of the vehicle withrespect to an expected traveling route on which the vehicle is totravel, or a yaw rate that is an amount of change in the yaw angle perunit time; predicting a possibility of deviation of the vehicle from theexpected traveling route, using at least one piece of information amongline-of-sight information about an occupant of the vehicle, utteranceinformation about the occupant, and traffic information; determiningdeviation of the vehicle from the expected traveling route, using theacquired yawing information, and the predicted possibility of deviation;and changing the superimposition target and correcting a difference inposition between the superimposition target after the change and thedisplay object, when the deviation of the vehicle from the expectedtraveling route is determined.